MASKED LANGUAGE MODELING

Pre-training Language Model as a Multi-perspective Course Learner (2023.findings-acl)

Challenge:	Pre-trained language models such as BERT are still poor in temporal reasoning . commonsense reasoning is crucial for natural language processing (NLP)
Approach:	They propose to use multi-step fine-tuning and masked language modeling to predict mangled temporal indicators that are crucial for commonsense reasoning.
Outcome:	The proposed model improves performance on multiple time-related tasks.

Copied to clipboard

Beiduo Chen, Shaohan Huang, Zihan Zhang, Wu Guo, Zhenhua Ling, Haizhen Huang, Furu Wei, Weiwei Deng, Qi Zhang

How does the pre-training objective affect what large language models learn about linguistic properties? (2022.acl-short)

Challenge:	Experimental results show that our method significantly improves ELECTRA’s average performance by 2.8% and 3.2% absolute points respectively on GLUE and SQuAD 2.0 benchmarks.
Approach:	They propose a multi-perspective course learning method to fetch many degrees and visual angles for sample-efficient pre-training and to fully leverage the relationship between generator and discriminator.
Outcome:	The proposed method improves ELECTRA's performance on GLUE and SQuAD 2.0 benchmarks and overshadows recent advanced ELECL-style models under the same settings.

Copied to clipboard

Ahmed Alajrami, Nikolaos Aletras

Improving the Lexical Ability of Pretrained Language Models for Unsupervised Neural Machine Translation (2021.naacl-main)

Challenge:	Several pre-training objectives have been proposed to pre-train language models . but, to our knowledge, no studies have investigated how different pre- training objectives affect what BERT learns about linguistic properties.
Approach:	They propose to use masked language modeling to pre-train language models . they propose to optimize a mangled language modeling objective to learn linguistic information .
Outcome:	The proposed objectives improve BERT's learning of linguistic properties compared to non-linguistically motivated objectives.

Copied to clipboard

Alexandra Chronopoulou, Dario Stojanovski, Alexander Fraser

Massive Choice, Ample Tasks (MaChAmp): A Toolkit for Multi-task Learning in NLP (2021.eacl-demos)

Challenge:	Existing methods for unsupervised neural machine translation (UNMT) use cross-lingual pretraining to align the lexical- and high-level representations of two languages.
Approach:	They propose to use type-level cross-lingual subword embeddings to enhance the bilingual masked language model pretraining with lexical-level information to align the two languages.
Outcome:	Empirical results show that the method improves on UNMT (up to 4.5 BLEU) and bilingual lexicon induction compared to baseline models.

Copied to clipboard

Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, Barbara Plank

Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages (2025.acl-srw)

Challenge:	Multi-task learning (MTL) has become a standard repertoire in natural language processing (NLP) it enables neural networks to learn tasks in parallel while leveraging the benefits of sharing parameters.
Approach:	They propose a toolkit for fine-tuning contextualized embeddings in multi-task settings.
Outcome:	The proposed toolkit supports a variety of natural language processing tasks . it enables neural networks to learn tasks in parallel while leveraging the benefits of sharing parameters.

Copied to clipboard

Daniil Gurgurov, Ivan Vykopal, Josef Van Genabith, Simon Ostermann

Choosing What to Mask: More Informed Masking for Multimodal Machine Translation (2023.acl-srw)

Challenge:	Low-resource languages (LRLs) face significant challenges in natural language processing due to limited data.
Approach:	They evaluate adapter-based methods for adapting mLMs to low-resource languages . they use unstructured text and structured knowledge from ConceptNet to evaluate adapters .
Outcome:	The proposed methods outperform large language models and LLaMA-3 and deepSeek-R1 models on low training data.

Copied to clipboard

Julia Sato, Helena Caseli, Lucia Specia

RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder (2022.emnlp-main)

Challenge:	Pre-trained language models have achieved remarkable results on several NLP tasks.
Approach:	They propose three new masking strategies for cross-lingual visual pre-training that focus on learning different linguistic patterns.
Outcome:	The proposed methods outperform the baseline model and achieve state-of-the-art accuracy on the Portuguese-English MMT task.

Copied to clipboard

Shitao Xiao, Zheng Liu, Yingxia Shao, Zhao Cao

Paraphrase-based Contrastive Learning for Sentence Pair Modeling (2025.naacl-srw)

Challenge:	Existing methods for dense retrieval are not effective, but there are still challenges.
Approach:	They propose a retrieval oriented pre-training paradigm based on Masked Auto-Encoder (MAE) where the sentence embedding is generated from the encoder’s masked input and the original sentence is recovered based upon the sentence embedded and decoded input via mangled language modeling.
Outcome:	The proposed model significantly improves the SOTA performance on a wide range of NLP benchmarks, like BEIR and MS MARCO.

Copied to clipboard

Seiji Sugiyama, Risa Kondo, Tomoyuki Kajiwara, Takashi Ninomiya

On-the-fly Cross-lingual Masking for Multilingual Pre-training (2023.acl-long)

Challenge:	Existing methods to improve performance of sentence pair modeling are not available on a large-scale for non-English languages.
Approach:	They propose a method to apply contrastive learning to pre-trained masked language models . they use sentence embeddings of paraphrase pairs to make similar sentences .
Outcome:	The proposed method can be used on four sentence pair modeling tasks in English and Japanese.

Copied to clipboard

Xi Ai, Bin Fang

Pairwise Prompt-Based Tuning with Parameter Efficient Fast Adaptation for Generalized Zero-Shot Intent Detection (2025.findings-naacl)

Challenge:	In multilingual pre-training, multilingual models only learn cross-linguality implicitly from isomorphic spaces formed by overlapping different language spaces due to the lack of explicit cross-linguistic forward pass.
Approach:	They propose a dynamic token-wise masking scheme for multilingual pre-training that uses a special token [C]x to replace a random token in the input sentence.
Outcome:	The proposed model improves the performance of UNMT models on De, Ro, Ne En.

Copied to clipboard

Xiaotong Zhang, Qianru Zhou, Han Liu, Hong Yu

PromptExplainer: Explaining Language Models through Prompt-based Learning (2024.findings-eacl)

Challenge:	Existing methods to generalize from seen intents to unseen intents are not effective . Xian et al., 2019: a novel approach to generalized zero-shot intent detection is needed .
Approach:	They propose a pairwise prompt-based tuning model with parameter efficient fast adaptation . they leverage hybrid contrastive learning in discriminant space and masked language modeling .
Outcome:	The proposed model can generalize to unseen intents with the help of seen intents . the proposed model is based on a pairwise prompt-based tuning model with fast adaptation .

Copied to clipboard

Zijian Feng, Hanzhang Zhou, Zixiao Zhu, Kezhi Mao

Language-agnostic BERT Sentence Embedding (2022.acl-long)

Challenge:	Existing explanation methods rely on linear approximations, accentuating irrelevant input tokens.
Approach:	They propose a method that aligns the explanation process with the masked language modeling task of pretrained language models and leverages prompt-based learning to generate class-dependent explanations.
Outcome:	Extensive experiments show that PromptExplainer outperforms state-of-the-art explanation methods.

Copied to clipboard

Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, Wei Wang

Improving Pretraining Techniques for Code-Switched NLP (2023.acl-long)

Challenge:	Existing methods for learning bilingual sentence embeddings are not well explored.
Approach:	They propose to combine best methods for learning multilingual sentence embeddings with pre-trained models to achieve 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba.
Outcome:	The proposed model achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, above the 65.5% achieved by LASER.

Copied to clipboard

Richeek Das, Sahasra Ranjan, Shreya Pathak, Preethi Jyothi

TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue (2020.emnlp-main)

Challenge:	Multilingual pretraining models for code-switched inputs are a key component of NLP applications.
Approach:	They propose to use masked language modeling techniques to mask code-switched text that are cognizant of language boundaries prior to masking.
Outcome:	The proposed techniques improve performance on two downstream tasks, Question Answering (QA) and Sentiment Analysis (SA), compared to standard pretraining techniques.

Copied to clipboard

Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, Caiming Xiong

Dialogue Systems & Conversational Agents

Challenge:	Existing pre-trained language models with self-attention encoder architectures are less useful in practice.
Approach:	They propose to use user and system tokens to model dialogue behavior during pre-training . they propose a contrastive objective function to simulate the response selection task .
Outcome:	The proposed model outperforms baseline models on four downstream tasks . it also has a few-shot ability that can mitigate the data scarcity problem .

DS-TOD: Efficient Domain Specialization for Task-Oriented Dialog (2022.findings-acl)

Copied to clipboard

Chia-Chien Hung, Anne Lauscher, Simone Ponzetto, Goran Glavaš

Protein-STORY: Semantic Text-Oriented Representation Yields biologically meaningful Protein embeddings (2026.acl-short)

Challenge:	Recent work shows that self-supervised dialog-specific pretraining on large conversational datasets yields substantial gains over traditional language modeling (LM) pretraining.
Approach:	They propose a resource-efficient and modular domain specialization by means of domain adapters in which domain knowledge is encoded.
Outcome:	The proposed framework extracts domain-specific terms and then uses them to build DomainCC and DomainReddit resources based on masked language modeling and response selection objectives.

Copied to clipboard

Nabil Ibtehaz, Daisuke Kihara

Representation Learning

Challenge:	Unsupervised representation learning relying on sequence data often overlooks decades of expert-curated biological knowledge stored in textual formats.
Approach:	They propose a pipeline that synthesizes protein embeddings from diverse, multi-source text descriptions and a network architecture that integrates high-fidelity functional and structural insights into a unified representation.
Outcome:	The proposed pipeline outperforms existing models on diverse downstream tasks (+2 pts F1) and enables zero-shot text-prompted protein search.

[MASK] Insertion: a robust method for anti-adversarial attacks (2023.findings-eacl)

Copied to clipboard

Xinrong Hu, Ce Xu, Junlong Ma, Zijian Huang, Jie Yang, Yi Guo, Johan Barthelemy

Leveraging Open Information Extraction for More Robust Domain Transfer of Event Trigger Detection (2024.findings-eacl)

Challenge:	Existing studies have focused on adversarial defenses against pretrained language models.
Approach:	They propose an adversarial defensing algorithm that inserts tokens into input sequences . they show an improvement in accuracy between 3.2 and 11.1 absolute points .
Outcome:	The proposed algorithm improves model accuracy on clean and polluted inputs compared with state-of-the-art models .

Copied to clipboard

David Dukić, Kiril Gashteovski, Goran Glavaš, Jan Snajder

Multilingual Normalization of Temporal Expressions with Masked Language Models (2023.eacl-main)

Challenge:	Event detection is a crucial information extraction task in many domains, such as Wikipedia or news.
Approach:	They propose to use subject-object relations obtained from a rule-based open information extraction system to enhance domain transfer of trigger detection (TD) they combine this enhanced transfer with masked language modeling on the target domain, observing further TD transfer gains.
Outcome:	The proposed model improves the transfer of triggers between domains and reduces performance drops when using a low-resource source domain to a high-res target domain.

Copied to clipboard

Lukas Lange, Jannik Strötgen, Heike Adel, Dietrich Klakow

Specializing Unsupervised Pretraining Models for Word-Level Semantic Similarity (2020.coling-main)

Challenge:	Existing methods for normalizing temporal expressions are rule-based, which severely limits the applicability in multilingual settings.
Approach:	They propose a neural method for normalizing temporal expressions based on masked language modeling and a slot-based prediction scheme for context-independent representations.
Outcome:	The proposed method outperforms existing rule-based methods in many languages and in particular, for low-resource languages with performance improvements of up to 33 F1 on average compared to the state of the art.

Copied to clipboard

Anne Lauscher, Ivan Vulić, Edoardo Maria Ponti, Anna Korhonen, Goran Glavaš

Composable Sparse Fine-Tuning for Cross-Lingual Transfer (2022.acl-long)

Challenge:	Unsupervised pretraining models encode only distributional knowledge encoded in text corpora, incorporated through language modeling objectives.
Approach:	They generalize a standard BERT model to a multi-task learning setting and integrate discrete knowledge on word-level semantic similarity into pretraining.
Outcome:	The proposed model outperforms the lexically blind “vanilla” model on several language understanding tasks.

Copied to clipboard

Alan Ansell, Edoardo Ponti, Anna Korhonen, Ivan Vulić

UEGP: Unified Expert-Guided Pre-training for Knowledge Rekindle (2024.findings-naacl)

Challenge:	Adapters and sparse fine-tuning have been developed to improve transfer learning . a number of approaches have been proposed to improve performance of fine-untuners .
Approach:	They propose a method that fine-tunes the entire set of parameters of a large pretrained model . they use adapters and sparse fine-uning to improve model efficiency .
Outcome:	The proposed method outperforms adapters in cross-lingual transfer benchmarks.

Copied to clipboard

Yutao Mou, Kexiang Wang, Jianhe Lin, Dehong Ma, Jun Fan, Daiting Shi, Zhicong Cheng, Gu Simiu, Dawei Yin, Weiran Xu

Bias Vector: Mitigating Biases in Language Models with Task Arithmetic Approach (2025.coling-main)

Challenge:	Existing paradigms for pre-training and fine-tuning have limitations . knowledge rekindle aims to break through performance upper bounds of experts without introducing additional annotated data.
Approach:	They propose a new paradigm for pre-training and fine-tuning that aims to re-incorporate the fine- tuned expert model into the training cycle and break through performance upper bounds of experts.
Outcome:	The proposed model breaks through performance upper bounds of experts without additional annotated data.

Copied to clipboard

Daiki Shirafuji, Makoto Takenaka, Shinya Taguchi

Take a Closer Look at Multilinguality! Improve Multilingual Pre-Training Using Monolingual Corpora Only (2023.findings-emnlp)

Challenge:	Using language models (LMs) has increased in use, and the use of biases and stereotypes is creating social problems.
Approach:	They propose a method to mitigate LM biases by continual training on biased data . they use masked language modeling to construct a Bias Vector as the difference between biased LMs and pre-trained LM weights .
Outcome:	The proposed method improves on the GLUE and SEAT benchmarks.

Copied to clipboard

Jinliang Lu, Yu Lu, Jiajun Zhang

Contextual Representation Learning beyond Masked Language Modeling (2022.acl-long)

Challenge:	Recent studies have demonstrated remarkable cross-lingual capability of pre-trained language models . however, semantic alignments may be the reason behind such capability but remain under-explored.
Approach:	They propose token-level and semantic-level code-switched masked language modeling to improve cross-lingual interactions over mono-mPLMs without parallel sentences.
Outcome:	The proposed method outperforms mono-mPLMs on natural language understanding and unsupervised machine translation tasks.

Copied to clipboard

Zhiyi Fu, Wangchunshu Zhou, Jingjing Xu, Hao Zhou, Lei Li

MOSAIC: Masked Objective with Selective Adaptation for In-domain Contrastive Learning (2026.findings-eacl)

Challenge:	masked language models adopt sampled embeddings as anchors to estimate and inject contextual semantics to representations.
Approach:	They propose a representation learning approach that uses embeddings as anchors to model contextual representations.
Outcome:	The proposed model achieves 5x speedup and 1.2 points average improvement over MLM.

Copied to clipboard

Vera Pavlova, Mohammed Makhlouf

A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models (2022.acl-long)

Challenge:	a new framework for domain adaptation of text embedding models addresses the challenges of adapting general-domain text embeds to specialized domains.
Approach:	They propose a framework for domain adaptation of text embedding models that integrates masked supervision and mangled objectives within a unified training pipeline.
Outcome:	The proposed framework improves on high-resource and low-resourced domains while preserving the robustness of the original model.

Copied to clipboard

Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, Xiang Ren

From Masked Language Modeling to Translation: Non-English Auxiliary Tasks Improve Zero-shot Spoken Language Understanding (2021.naacl-main)

Challenge:	Recent few-shot learning models such as GPT3 are expensive and slow to deploy for real-world applications.
Approach:	They propose a prompt-based low-resource learning method for VL tasks with a few examples . they pre-train a sequence-to-sequence transformer model with prefix and masked language modeling .
Outcome:	The proposed method outperforms Frozen on vision-language tasks with prompt-based learning by 18.2% point.

Copied to clipboard

Rob van der Goot, Ibrahim Sharaf, Aizhan Imankulova, Ahmet Üstün, Marija Stepanović, Alan Ramponi, Siti Oryza Khairunnisa, Mamoru Komachi, Barbara Plank

SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity Representation (2022.findings-emnlp)

Challenge:	Lack of publicly available evaluation data for low-resource languages limits progress in SLU . despite advances in neural modeling for slot and intent detection, datasets for SLU remain limited.
Approach:	They propose a joint learning approach with English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer.
Outcome:	The proposed model can learn English SLU training data and non-English auxiliary tasks from raw text, syntax and translation for transfer.

Copied to clipboard

Zekun Li, Jina Kim, Yao-Yi Chiang, Muhao Chen

Curriculum Masking in Vision-Language Pretraining to Maximize Cross Modal Interaction (2024.naacl-long)

Challenge:	Named geographic entities are the building blocks of many geographic datasets.
Approach:	They propose a spatial language model that provides a general-purpose geo-entity representation based on neighboring entities in geospatial data.
Outcome:	The proposed model improves on two downstream tasks, showing significant performance improvement compared with existing models that do not use spatial context.

Copied to clipboard

Kraig Tou, Zijun Sun

Multimodality

Challenge:	masked language modeling is widely used as a pretraining component in Vision and language (V+L) but performance on benchmarks has not received the attention it deserves.
Approach:	They propose a curriculum masking scheme that uses a parallel mask selection agent to mask tokens at a frequency proportional to the level of cross modal interaction necessary to reconstruct them.
Outcome:	The proposed method improves relational understanding on a wide range of V+L tasks.

Bidirectional Transformer Reranker for Grammatical Error Correction (2023.findings-acl)

Copied to clipboard

Ying Zhang, Hidetaka Kamigaito, Manabu Okumura

Enabling Autoregressive Models to Fill In Masked Tokens (2026.findings-eacl)

Challenge:	Pre-trained seq2seq models suffer from a prediction bias due to their unidirectional decoding.
Approach:	They propose a bidirectional Transformer reranker that re-estimates the probability of each candidate sentence generated by pre-trained seq2seq models.
Outcome:	The proposed model improves on the original model and gives a 59.52 GLEU score on the JFLEG corpus.

Copied to clipboard

Daniel Mingyi Israel, Aditya Grover, Guy Van den Broeck

Token Dropping for Efficient BERT Pretraining (2022.acl-long)

Challenge:	Autoregressive (AR) and masked language modeling (MLM) models are incapable of mucked infilling, which is the ability to predict mangled tokens between past and future context.
Approach:	They propose a method that leverages the strengths of autoregressive and masked language modeling to achieve state-of-the-art mucked infilling performance.
Outcome:	The proposed approach outperforms existing methods on masked infilling tasks.

Copied to clipboard

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, Denny Zhou

Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment (2021.acl-long)

Challenge:	Existing methods to accelerate pretraining of transformer-based models are computationally expensive and degrade performance on downstream tasks.
Approach:	They propose a "token dropping" method to accelerate the pretraining of transformer-based models by 25% . they leverage the already built-in masked language modeling loss to identify unimportant tokens with practically no computational overhead.
Outcome:	The proposed method reduces the pretraining cost of BERT models by 25% while achieving similar overall performance on downstream tasks.

Copied to clipboard

Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei

Modeling Mathematical Notation Semantics in Academic Papers (2021.findings-emnlp)

Challenge:	Experimental results show that denoising word alignment improves cross-lingual transferability . most applications and resources are still English-centric, making non-English users hard to access.
Approach:	They propose to denoise word alignment as a cross-lingual pre-training task . they first self-label word alignments for parallel sentences and then mask tokens .
Outcome:	The proposed model improves cross-lingual transferability on token-level tasks, especially on question answering, and structured prediction.

Copied to clipboard

Hwiyeol Jo, Dongyeop Kang, Andrew Head, Marti A. Hearst

GeoLM: Empowering Language Models for Geospatially Grounded Language Understanding (2023.emnlp-main)

Challenge:	Existing models that can predict mathematical notation are unable to analyze mathematical notations reliably.
Approach:	They propose two tasks that can be used to train a model that selectively masks notation tokens and encodes left and/or right sentences as context.
Outcome:	The proposed model performs better than baseline models trained by masked language modeling compared to baseline models, but is less accurate than token-level models .

Copied to clipboard

Zekun Li, Wenxuan Zhou, Yao-Yi Chiang, Muhao Chen

Causal Distillation for Language Models (2022.naacl-main)

Challenge:	Pretrained language models do not utilize valuable geospatial information in large databases, e.g., OpenStreetMap.
Approach:	They propose a geospatially grounded language model that connects linguistic and geospheric contexts.
Outcome:	The proposed model bridges the gap between natural language processing and geospatial sciences.

Copied to clipboard

Zhengxuan Wu, Atticus Geiger, Joshua Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christopher Potts, Noah Goodman

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations (2021.emnlp-main)

Challenge:	Distillation efforts have led to language models that are more compact and efficient without serious drops in performance.
Approach:	They propose to augment distillation with a third objective that encourages the student model to imitate the causal dynamics of the teacher through a distillation interchange intervention training objective (DIITO).
Outcome:	The proposed method lowers perplexity on the WikiText-103M corpus and improves on the GLUE benchmark, SQuAD, and CoNLL-2003.

Copied to clipboard

Hang Li, Wenbiao Ding, Yu Kang, Tianqiao Liu, Zhongqin Wu, Zitao Liu

Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale. (2023.findings-acl)

Challenge:	Existing audio-language task-specific predictive approaches focus on building complicated late-fusion mechanisms.
Approach:	They propose a cross-modal transformer for audio-and-language that learns inter-modal connections between audio and language through two proxy tasks on a large amount of audio- and-language pairs.
Outcome:	The proposed model improves on multiple audio-and-language tasks and can be used in fine-tuning phase.

Copied to clipboard

Vijeta Deshpande, Dan Pechi, Shree Thatte, Vladislav Lialin, Anna Rumshisky

Unsupervised Dependency Graph Network (2022.acl-long)

Challenge:	Recent studies have focused on high-compute settings, leaving the question of when these abilities begin to emerge largely unanswered.
Approach:	They investigate whether effects of pre-training can be observed when problem size is reduced, modeling a smaller, reduced-vocabulary language.
Outcome:	The proposed model performance is correlated with pre-training perplexity and performance.

Copied to clipboard

Yikang Shen, Shawn Tan, Alessandro Sordoni, Peng Li, Jie Zhou, Aaron Courville

MERMAID: Metaphor Generation with Symbolism and Discriminative Decoding (2021.naacl-main)

Challenge:	Recent work has identified properties of pretrained self-attention models that mirror those of dependency parse structures.
Approach:	They propose a model that encourages attention heads to model different dependency relations from raw corpora and a masked language modeling task.
Outcome:	The proposed model can induce dependency structures from raw corpora and the masked language modeling task without gold POS tags and any external information.

Copied to clipboard

Tuhin Chakrabarty, Xurui Zhang, Smaranda Muresan, Nanyun Peng

Commonsense Reasoning

Challenge:	a new method for generating metaphors is proposed to generate literal sentences . human evaluations show that our best model generates metaphors better than three well-crafted baselines 66% of the time on average.
Approach:	They propose a method to automatically construct a parallel corpus by transforming literal sentences to metaphorical ones using commonsense inference and masked language modeling.
Outcome:	The proposed method generates metaphors better than baselines 66% of the time on average.

Patton: Language Model Pretraining on Text-Rich Networks (2023.acl-long)

Copied to clipboard

Bowen Jin, Wentao Zhang, Yu Zhang, Yu Meng, Xinyang Zhang, Qi Zhu, Jiawei Han

Learning to Sample Replacements for ELECTRA Pre-Training (2021.findings-acl)

Challenge:	Existing models for text-rich networks do not take inter-document structure into account.
Approach:	They propose a pretraining framework for a text-rich network using a masked language model and a masking node prediction framework.
Outcome:	The proposed model outperforms baselines on four tasks in academic and e-commerce domains.

Copied to clipboard

Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei

Taking Actions Separately: A Bidirectionally-Adaptive Transfer Learning Method for Low-Resource Neural Machine Translation (2022.coling-1)

Challenge:	Experimental results show that ELECTRA pretrains a discriminator to detect replaced tokens . despite compelling performance, there is no direct feedback loop from discriminator and generator to generator, making replacements biased to correct tokens.
Approach:	They propose to augment sampling with a hardness prediction mechanism to encourage the discriminator to learn what it has not acquired.
Outcome:	The proposed method improves ELECTRA pre-training on various downstream tasks.

Copied to clipboard

Xiaolin Xing, Yu Hong, Minhan Xu, Jianmin Yao, Guodong Zhou

Contrastive Learning for Prompt-based Few-shot Language Learners (2022.naacl-main)

Challenge:	Existing approaches to train NMT models rely on sparse parallel data . a variety of PC variants yield significant improvements for low-resource NMT .
Approach:	They propose to transfer well-trained NMT models to low-resource languages by bidirectionally-adaptive learning strategy . they divide inner constituents of Parent encoder into two "teams" aiming to adapt to characteristics of low- and high-resourced languages .
Outcome:	The proposed method improves on low-resource NMT models with a variety of PC variants.

Copied to clipboard

Yiren Jian, Chongyang Gao, Soroush Vosoughi

SCRIPT: Self-Critic PreTraining of Transformers (2021.naacl-main)

Challenge:	a recent study has shown that GPT-3 fine-tuning models with limited examples is effective . a contrastive learning framework clusters inputs from the same class under different augmented “views” and repels those from different classes.
Approach:	They propose a supervised contrastive framework that clusters inputs from the same class under different augmented "views" they combine a contrastive loss with the standard masked language modeling loss in prompt-based few-shot learners .
Outcome:	The proposed framework improves on the state-of-the-art methods in a diverse set of 15 language tasks.

Copied to clipboard

Erik Nijkamp, Bo Pang, Ying Nian Wu, Caiming Xiong

A Closer Look at Parameter Contributions When Training Neural Language and Translation Models (2022.coling-1)

Challenge:	Existing methods for representation learning of text are masked language modeling (MLM) a language model is trained to learn universal contextual embeddings, which are fine-tuned on a down-stream task.
Approach:	They propose a self-critic pretraining transformer for representation learning of text . they demonstrate improved sample-efficiency and improved performance over strong baselines .
Outcome:	The proposed model improves sample-efficiency and performance over strong baselines.

Copied to clipboard

Raúl Vázquez, Hande Celikkanat, Vinit Ravishankar, Mathias Creutz, Jörg Tiedemann

Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling (D19-1)

Challenge:	Neural models and Transformers have been used for almost every NLP task . however, the intrinsic dynamics of the training procedure have not been studied in depth for highly complex network architectures.
Approach:	They analyze the learning dynamics of neural language and translation models using Loss Change Allocation indicator . they use a standard Transformer architecture to train a model with three learning objectives .
Outcome:	The proposed model is based on a standard model that is used for training tasks.

Copied to clipboard

Xiaochuang Han, Jacob Eisenstein

Phrase-aware Unsupervised Constituency Parsing (2022.acl-long)

Challenge:	Contextualized word embeddings are becoming a ubiquitous component of natural language processing.
Approach:	They propose a domain-adaptive fine-tuning approach to pretrain on unlabeled text . they test this approach on sequence labeling in two challenging domains .
Outcome:	The proposed approach improves on sequence labeling in two domains: Early Modern English and Twitter.

Copied to clipboard

Xiaotao Gu, Yikang Shen, Jiaming Shen, Jingbo Shang, Jiawei Han

Low-Resource NLP

Challenge:	Recent studies have achieved inspiring success in unsupervised grammar induction using masked language modeling (MLM) as the proxy task.
Approach:	They propose to regularize the parser with phrases extracted by an unsupervised phrase tagger to help the LM model quickly manage low-level structures.
Outcome:	The proposed method improves the identification of high-level structures using phrase-guided masking.

The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives (D19-1)

Copied to clipboard

Elena Voita, Rico Sennrich, Ivan Titov

Prompt-learning for Fine-grained Entity Typing (2022.findings-emnlp)

Challenge:	a recent study has shown that deep neural networks are effective with various tasks . a new study examines how representations of tokens evolve between layers under different learning objectives .
Approach:	They use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers.
Outcome:	The proposed model outperforms untrained models on word identity prediction tasks . the model outpersforms models trained on other linguistic tasks based on the model's objective .

Copied to clipboard

Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Xiaobin Wang, Pengjun Xie, Haitao Zheng, Zhiyuan Liu, Juanzi Li, Hong-Gee Kim

LXMERT: Learning Cross-Modality Encoder Representations from Transformers (D19-1)

Challenge:	Extensive experiments on fine-grained entity typing under fully supervised, few-shot, and zero-shot settings show the effectiveness of prompt-learning.
Approach:	They propose a prompt-learning pipeline that stimulates versatile knowledge of pre-trained language models (PLMs) by constructing entity-oriented verbalizers and templates and conducting masked language modeling.
Outcome:	The proposed approach can be applied to fine-grained entity typing in fully supervised, few-shot, and zero-shot scenarios.

Copied to clipboard

Hao Tan, Mohit Bansal

LinkBERT: Pretraining Language Models with Document Links (2022.acl-long)

Challenge:	Existing models with better representations of visual content and language have been developed for visual-content understanding.
Approach:	They propose a framework to learn vision-and-language connections from Transformers models . they pre-train a large-scale Transformer model with large amounts of image-and sentence pairs .
Outcome:	The proposed model improves state-of-the-art on two visual-reasoning tasks by 22% . the proposed model is based on a large-scale Transformer model with three encoders .

Copied to clipboard

Michihiro Yasunaga, Jure Leskovec, Percy Liang

StructFormer: Joint Unsupervised Induction of Dependency and Constituency Structure from Masked Language Modeling (2021.acl-long)

Challenge:	Existing language model pretraining methods do not capture dependencies or knowledge that span across documents.
Approach:	They propose a language model pretraining method that leverages links between documents . they use masked language modeling and document relation prediction to model LMs .
Outcome:	The proposed method outperforms existing methods on downstream tasks across two domains.

Copied to clipboard

Yikang Shen, Yi Tay, Che Zheng, Dara Bahri, Donald Metzler, Aaron Courville

Train No Evil: Selective Masking for Task-Guided Pre-Training (2020.emnlp-main)

Challenge:	Existing models that induce grammar structures from data focus on constituency or dependency structures alone.
Approach:	They propose a model that can induce dependency and constituency structure at the same time.
Outcome:	The proposed model can induce both constituency and dependency structures at the same time.

Copied to clipboard

Yuxian Gu, Zhengyan Zhang, Xiaozhi Wang, Zhiyuan Liu, Maosong Sun

On the Influence of Masking Policies in Intermediate Pre-training (2021.emnlp-main)

Challenge:	Pre-trained language models can't capture domain-specific and task-specific patterns because of the task-agnostic pre-training stage.
Approach:	They propose a task-guided pre-training stage with selective masking between general pre-train and fine-tuning to learn domain-specific patterns.
Outcome:	The proposed method can achieve comparable or even better performance with less than 50% of computation cost.

Copied to clipboard

Qinyuan Ye, Belinda Z. Li, Sinong Wang, Benjamin Bolte, Hao Ma, Wen-tau Yih, Xiang Ren, Madian Khabsa

Developing Language Resources and NLP Tools for the North Korean Language (2022.lrec-1)

Challenge:	Existing studies show that inserting an intermediate pre-training stage improves performance of masked language models.
Approach:	They propose methods to automate the discovery of optimal masking policies via direct supervision or meta-learning.
Outcome:	The proposed method outperforms the heuristic of masking named entities on TriviaQA and can be generalizable beyond that task.

Copied to clipboard

Arda Akdemir, Yeojoo Jeon, Tetsuo Shibuya

A Pilot Study on Dialogue-Level Dependency Parsing for Chinese (2023.findings-acl)

Challenge:	There are no linguistic sources for the North Korean language, resulting in a lack of a Korean language model.
Approach:	They present a large-scale dataset for the North Korean language and annotate a subset of this dataset for a sentiment analysis task.
Outcome:	The proposed model performs better than other models for masked language modeling and sentiment analysis tasks.

Copied to clipboard

Gongyao Jiang, Shuang Liu, Meishan Zhang, Min Zhang

Syntactic Text Processing

Challenge:	Dialogue-level dependency parsing has received insufficient attention, especially for Chinese.
Approach:	They propose a signal-based method to transform seen syntactic dependencies into unseen ones between elementary discourse units (EDUs) they apply single-view and multi-view data selection to access reliable pseudo-labeled instances.
Outcome:	The proposed method transforms seen syntactic dependencies into unseen ones between elementary discourse units (EDUs) the proposed method also provides reliable pseudo-labeled instances.

The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining (2023.emnlp-main)

Copied to clipboard

Ting-Rui Chiang, Dani Yogatama

Generating Multiple-choice Questions for Medical Question Answering with Distractors and Cue-masking (2024.lrec-main)

Challenge:	Despite the rise of the prompting paradigm with the scaling breakthrough of very large language models, understanding the mechanism of model fine-tuning remains an important endeavor.
Approach:	They analyze the masked language modeling pretraining objective function from the perspective of the Distributional Hypothesis and examine whether the distributional property leads to better sample efficiency and better generalization capability of pretrained models.
Outcome:	The proposed model pretraining objective function improves sample efficiency and generalization capability but does not explain the generalization ability of natural language models.

Copied to clipboard

Damien Sileo, Kanimozhi Uma, Marie-Francine Moens

Contextualized Semantic Distance between Highly Overlapped Texts (2023.findings-acl)

Challenge:	Medical multiple-choice question answering (MCQA) requires high accuracy to be useful in practice.
Approach:	They propose to focus masked language modeling on disease name prediction when using medical encyclopedic paragraphs as input.
Outcome:	The proposed model outperforms the masked language model on disease name prediction and masks the cues to the answers.

Copied to clipboard

Letian Peng, Zuchao Li, Hai Zhao

AxomiyaBERTa: A Phonologically-aware Transformer Model for Assamese (2023.findings-acl)

Challenge:	Conventional semantic metrics are based on word representations and are vulnerable to disturbance of overlapped components with similar representations.
Approach:	They propose a mask-and-predict strategy to evaluate the semantic distance between the overlapped sentences using words in the longest common sequence as neighboring words and use masked language modeling to predict their positions.
Outcome:	The proposed method outperforms the state-of-the-art in domain adaption by a huge margin.

Copied to clipboard

Abhijnan Nath, Sheikh Mannan, Nikhil Krishnaswamy

IAD: In-Context Learning Ability Decoupler of Large Language Models in Meta-Training (2024.lrec-main)

Challenge:	AxomiyaBERTa is a novel BERT model for low-resource languages . Transformers require extensive computing resources and suffer in low-compute settings .
Approach:	They propose a novel BERT model for Assamese, a morphologically-rich low-resource language of eastern India that is trained on a simple masked language modeling task without the NSP objective.
Outcome:	The proposed model performs well on token-level tasks and on “longer context” tasks with the aid of embedding disperser and phonological signals.

Copied to clipboard

Yuhan Liu, Xiuying Chen, Gao Xing, Ji Zhang, Rui Yan

CLEVR-Implicit: A Diagnostic Dataset for Implicit Reasoning in Referring Expression Comprehension (2023.emnlp-main)

Challenge:	In-context Learning (ICL) is a paradigm in which LLMs acquire task-specific knowledge by processing input-output pairs provided as prompts.
Approach:	They propose an In-context learning Ability Decoupler to separate ICL ability from general ability of LLMs in meta-training phase . they first identify parameters suitable for ICL by transference-driven gradient importance and propose a new max-margin loss to emphasize the separation of the two abilities.
Outcome:	The proposed model separates the ICL ability from the general ability of LLMs in the meta-training phase, where the I-related parameters are tuned to adapt for ICL tasks.

Copied to clipboard

Jingwei Zhang, Xin Wu, Yi Cai

Looking Right is Sometimes Right: Investigating the Capabilities of Decoder-only LLMs for Sequence Labeling (2024.findings-acl)

Challenge:	Recent studies show that pre-trained vision-language models perform well in cross-modal tasks, including referring expression comprehension.
Approach:	They propose a method that enables VL models to reason with implicit text . they propose to use a dataset to align the text with objects in the images .
Outcome:	The proposed method improves performance 37.94% on referring expression comprehension task.

Copied to clipboard

David Dukić, Jan Snajder

Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling (2023.findings-acl)

Challenge:	Pre-trained language models excel in natural language understanding (NLU) tasks.
Approach:	They propose to apply layer-dependent removal of the causal mask (CM) during LLM fine-tuning to improve SL performance.
Outcome:	The proposed approach outperforms state-of-the-art SL models on IE tasks, while achieving state- of-the art results is unclear.

Copied to clipboard

Gábor Berend

Do Transformers Parse while Predicting the Masked Word? (2023.emnlp-main)

Challenge:	a recent study suggests that masked language models are a useful pre-training technique for natural language processing . a study using mlms pre-trained by a team of researchers has improved performance .
Approach:	They propose an alternative to the classic masked language modeling paradigm . they use an unsupervised technique which uses sparse coding to make the prediction possible .
Outcome:	The proposed technique improves on pre-trained models compared to vanilla MLM . the proposed model returns distributions over their vocabulary peaking at plausible substitutes .

Copied to clipboard

Haoyu Zhao, Abhishek Panigrahi, Rong Ge, Sanjeev Arora

Extrapolating Multilingual Understanding Models as Multilingual Generators (2023.findings-emnlp)

Challenge:	Existing studies show that pre-trained language models encode linguistic structures like parse trees while being trained unsupervised.
Approach:	They propose to train pre-trained language models to encode linguistic structures like parse trees while unsupervised.
Outcome:	The proposed model performs optimally for masked language modeling loss on the English PCFG.

Copied to clipboard

Bohong Wu, Fei Yuan, Hai Zhao, Lei Li, Jingjing Xu

BERTwich: Extending BERT’s Capabilities to Model Dialectal and Noisy Text (2023.findings-emnlp)

Challenge:	Existing multilingual understanding models are not capable of generating high-quality text compared with decoder-based causal language models.
Approach:	They propose a method to adapt a multilingual encoder to a language generator with a small number of additional parameters.
Outcome:	The proposed approach outperforms initialization-based methods with 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation.

Copied to clipboard

Aarohi Srivastava, David Chiang

Leveraging External Knowledge for Historical Document Restoration via Retrieval-Augmented Large Language Models (2026.findings-acl)

Challenge:	Pre-trained language models like BERT deteriorate in the face of dialect variation or noise.
Approach:	They propose to sandwich BERT's encoder stack between additional encoder layers trained to perform masked language modeling on noisy text.
Outcome:	The proposed approach promotes zero-shot transfer to dialectal text and reduces embedding space between words and noisy counterparts.

Copied to clipboard

Gabeen Kim, Kyeongpil Kang